Adaptive constraints by morphological operations for single-shot digital holography

Digital holography provides access to quantitative measurement of the entire complex field, which is indispensable for the investigation of wave-matter interactions. The emerging iterative phase retrieval approach enables to solve the inverse imaging problem only from the given intensity measurements and physical constraints. However, enforcing imprecise constraints limits the reconstruction accuracy and convergence speed. Here, we propose an advanced iterative phase retrieval framework for single-shot in-line digital holography that incorporates adaptive constraints, which achieves optimized convergence behavior, high-fidelity and twin-image-free reconstruction. In conjunction with morphological operations which can extract the object structure while eliminating the irrelevant part such as artifacts and noise, adaptive constraints allow the support region to be accurately estimated and automatically updated at each iteration. Numerical reconstruction of complex-valued objects and the capability of noise immunity are investigated. The improved reconstruction performance of this approach is experimentally validated. Such flexible and versatile framework has promising applications in biomedicine, X-ray coherent diffractive imaging and wavefront sensing.

www.nature.com/scientificreports/ pioneering research work of iterative phase retrieval dates back to Gerchberg-Saxton (GS) algorithms which is slow and sensitive to initial guesses 32 , therefore inspiring a furry of follow-up work to improve the GS iterative framework [33][34][35] . Such GS-based algorithms exploit back-and-forth propagation between different planes, embedding physical constraints like support [36][37][38] , non-negativity 39,40 , absorption 21 into iteration to speed up convergence and avoid local convergence during iteration. Support constraints require the prior knowledge of the object shape, so as to set the transmission function of the object beyond the known object boundary to zero. To escape the need of such prior knowledge, the adaptive support of "shrink-wrap" technique based in the spatial 37 and frequency 38 domain estimates and updates the support area according to the range where the intensity is over a certain threshold. However, if the determined threshold level is lower, the support region could be overestimated which induces the residue of noise in the reconstruction. While if the determined threshold level is higher, some object information could be missing. The inaccurate selection of the threshold causes a less than perfect estimation of the support. Thus, the absorption constraint has been proposed relying on the principle that absorption may not give rise to an increased amplitude following a scattering process. Although such effective method dispenses with the object support, it is limited to the samples which do not absorb or scatter optical wave significantly 41 . Recent researches favor multi-image phase retrieval because it is flexible to the issues existing in the GS-based algorithms. Multiple frames of raw images serve as amplitude constraints to force the complex object field to gradually agree well with all these measurements. The data redundancy supplied by multiple measurements is capable of reconstructing complex-valued objects 42 . Multiple measured holograms can be achieved by varying object-to-sensor distances 38,[43][44][45] , illumination wavelengths 46,47 , illumination angles 48,49 and coding mask modulation 50 . Unfortunately, the achievable of a stable and accurate reconstruction is strongly enslaved to high-precision controllable devices, slow convergence rate, great quantities of holographic data and iterations. Hence, optimizing the phase retrieval approach for high-quality reconstruction and improved noise immunity still remains a challenging task. Morphological filtering technique is a nonlinear signal processing method stemming from set theory and integral geometry 51 , which is developed to numerous applications such as biomedical image processing, machined surface inspection and fault diagnosis. Typically, morphological operations, comprising erosion, dilation, closing, opening, related combined and compound operations, can be employed to modify the geometry of the raw image, thereby to realize the extraction of the exact pictorial information. In this work, we propose an advanced iterative phase retrieval method that generates adaptive constraints by morphological operators for single-shot in-line digital holography to improve the reconstruction quality and speed up convergence. In this proposed scheme, the square root of a single intensity pattern is used to update the modulus of the diffracted wave field in the diffraction space, and the adaptive constraint updated iteratively by morphological operators is applied to confine the object wave field, so that it can adaptively tend to the accurate amplitude and phase distributions of a complex object and accelerate the convergence speed. Compared to previous strategies for generating the object support, the proposed approach in conjunction with morphological operations provides a more appropriate and efficient estimation of support. The support region generated automatically is tight enough to eliminate the need of a sequence of intensity patterns. The method of generating adaptive constraints by morphological operations is described in detail. The numerical calculation and the immunity to noise are discussed. Moreover, the experimental reconstructions are presented to verify the stability and accuracy of our approach.

Methods
Optical setup for in-line digital holographic imaging. Here, an in-line digital holographic imaging configuration is considered, which is presented in Fig. 1a. A linearly polarized Gaussian laser beam is emitted from a He-Ne laser at the wavelength of 632.8 nm. After being collimated and expanded by the beam expander (BE) which is comprised of two lenses, the laser beam passes through the object. Then the interference of the scattered and unscattered wave fields generates an in-line hologram recorded by a CMOS sensor (QHY174, QHYCCD Co., Ltd.). The CMOS sensor we used has the resolution of 1200 × 1920 with a pixel size of 5.86 µ m, and the field of view (FOV) of such holographic imaging system is 7.032 × 11.251 mm 2 . On account of the limited FOV of the CMOS, the diagram acts as a low-pass filter to select a certain area of the object to image on the CMOS. Such lensless in-line digital holographic imaging system is characterized by portability, low cost and high space-bandwidth product.
Principles of adaptive constraints applied in iterative phase retrieval. Mathematically, the phaseretrieval problem can be formulized as 52 where u ∈ C is the complex-valued signal representing the transmission of the interested object, I is the measured intensity pattern, A expresses the wavefront propagation process, ⊙ refers to the Hadamard product, M represents the modulated optical masks that provides constraints to optimize the convergence and reduce the ill-posedness of the inverse problem. Early researches concentrated on introducing support constraints which limit the reconstructed region into the double-side constraint iteration. Specially, the mathematical definition of the support constraint describes as u i = u i i ∈ S 0 i / ∈ S , where S denotes the set of pixels within the support region. The conventional backpropagation reconstruction overlapped with twin image is presented in Fig. 1b1. In the iterative phase retrieval method, the support constraint is commonly imposed on the object domain. Given the prior knowledge of the object size, the support constraint is generally assumed as a rectangular or circular aperture to eliminate the information located outside the estimated support region 53 . As seen in Fig. 1b2, such loose and rough support www.nature.com/scientificreports/ enables to suppress artifacts outside the boundaries, but artifacts distributed within the support region still preserve, which significantly obfuscates the reconstruction. Moreover, in terms of the target object in a particular shape, designing a specific support is complicated and inefficient. Hence, morphological operations are introduced to generate adaptive constraints, aiming to improve the accuracy and efficiency of extracting the relevant image structures and solve the twin-image problem. Morphological filtering, mainly involving morphological operations and structural element (SE), is regarded as a powerful image processing tool for analyzing and extracting the geometrical structure of an image even wrapped by noise. The interactions between a raw image and a certain pre-designed SE can be investigated by morphological operations 51 . A binary SE represents a smaller matrix of pixels than that of the raw image, whose features are determined by its shape and domain. Mathematically, the binary erosion of a set F by a pre-defined SE (G) is expressed as 54 where represents the erosion operation, (G) ε = σ |σ = g + ε, g ∈ G denotes the translation of origin of G to point ε . It means that erosion replaces each pixel with the local minimum of all pixels in the neighborhood www.nature.com/scientificreports/ whose shape and size is determined by SE. On the contrary, the binary dilation replaces each pixel with the local maximum of all pixels in the neighborhood, which is defined as 54 where ⊕ represents the dilation operation, Ĝ is the reflection of G. Based on the combination of erode and dilation operations, the opening and closing operations can be described respectively as where • denotes the opening operation and • denotes the closing operation. Morphological opening operation eliminates the irrelevant part such as scatters and burrs in the image while preserving the primary geometry of the object structure. Morphological closing operation fills the small holes and thickens the foreground pixels in the image. Because the morphological opening and closing functions are not mutually inverse 55 , they can be further cascaded as open-closing and close-opening operations by using different SE (G1, G2), which can be considered as On the basis of the skeletons and morphological characteristics in an image, morphological operations have the ability to selectively remove the unwanted structures such as noise and unrelated target objects. The reconstruction shown in Fig. 1b3 indicates that using adaptive constraints obviously enhances extraction capability and removes disturbance of twin-image artifacts. It offers a more appropriate and efficient estimation of object support. Stages for the generation of adaptive constraints are illustrated in Fig. 1c. An input image U is firstly binarized using Poisson distribution-based threshold 56 . Such automatic threshold algorithm dispenses with the need for setting a certain threshold and optimizing any parameters. Then the morphological adjustments are considered for refining the image. Due to the shrinkage of the closing operator and the expansibility of the opening operator 55 , close-opening filter enlarging the object domain incurs the inadequate removal of noise, while open-closing filter contracting the object domain causes the partial loss of relevant image information. To get the trade-off between these two filters, an average weighted combination of open-closing and close-opening is calculated. After undergoing morphological operations, Gaussian filtering is applied to smooth the image F . Finally, the adaptive constraint is formed upon binarizing the filtering image again according to the threshold obtained by edge detection method 'Sobel' . If the constraint is created by using the threshold-based segmentation algorithm in a straightforward manner, the binary pattern depends on the area where the intensity is above the pre-assessed threshold. In this case, the accuracy of estimating the object support is susceptible to twin-image artifacts and measurement noise in experiment. Instead, morphological filtering enables to extract the geometrical structure of featured objects even in cases involving experimental noise, utilizing a SE to probe each pixel and modify its grayscale value according to the intensity of all pixels in its neighborhood. Consequently, the adaptive constraint generated by morphological filtering is updated at each iteration, giving rise to a sharper and tighter support region.
In pursuit of an advanced iterative phase retrieval method for in-line hologram that endows with highfidelity reconstruction, superior convergence speed and improved immunity to noise, adaptive constraints by morphological operations are introduced to the iterative phase retrieval algorithm. The overview flowchart is depicted in Fig. 2a. The iterative process between the sensor plane and the object plane is performed as follows.
• Step 1: Initialization. The in-line hologram with M 1 × N 1 pixels captured by the CMOS is padded with The initial amplitude is identified as the square root of the padding hologram, and the initial phase ϕ 0 is set to random or constant. Subsequently, the initial complex-valued wave field is expressed as U k s = √ I 0 exp jϕ 0 . • Step 2: The process of the wave field propagating in the free space is calculated by the angular spectrum method (ASM) 57 , which takes the form as where F and F −1 represent two-dimensional Fourier transform and inverse Fourier transform respectively, H is the transfer function defined as where (f x , f y ) is the coordinate in spatial frequency domain, z denotes the distance from the object plane to the sensor plane. In digital holography, there are various types of reconstruction noise that can affect the quality and accuracy of the reconstruction. In order to achieve a better reconstruction, it is important to acquire the focus distance of an object. The optimal reconstruction plane can be identified based on Tamura coefficient (TC) metric which has only a single extreme in the whole focus range 58 . Hence, taking advantage of the intrinsic flexibility of digital holography due to its numerical focusing ability, multiple sections of an www.nature.com/scientificreports/ object can be reconstructed by locating the best focus distance corresponding to different sections of the object. is the wavelength. The complex-amplitude field U k s distributed in the sensor plane is propagated back to the object plane to obtain Step 3: In the object plane, the adaptive constraint (AC) is implemented to update both the amplitude A k o and the phase ψ k o . The generation of adaptive constraints combines thresholding technique and morphological operations, which is illustrated in Fig. 1c in detail. Steps 2-4 are processed repeatedly until the m th iteration. Finally the reconstruction can be achieved by back propagating from the sensor plane to the object plane. The free-space propagation is calculated as a circular convolution model via fast Fourier transforms(FFTs), which causes wraparounds superimposed on the calculation result. In order to resolve this issue, adequate padding is used to prevent periodization artifacts and obtain an accurate calculation 59 . Evolutions of adaptive constrains enforced on the amplitude and the phase in the object plane are exhibited in Fig. 2b. The adaptive constraints are generated at each iteration by employing morphological operations based on the reconstructed amplitude and phase in the object plane respectively. The implementation of morphological filtering helps to extract the object structure while simultaneously eliminating the artifact and noise. As the iteration process continues, the constraint is updated automatically and performs a Hadamard product with the reconstructed images in the object plane. Hence, irrelevant regions are gradually filtered out, causing the support region to automatically shrink to match the geometrical structure of the object. The generation of adequate tight and sharp support region is conducive to promote the accuracy of extracting the object information and suppress the overlapping twin-image artifact that obscures the reconstruction.

Numerical calculation
Reconstruction of complex-valued objects. For the purpose of verifying the improvement effect of employing adaptive constraints in the iterative phase retrieval, the comparison of reconstruction quality and convergence behavior with multi-distance phase retrieval (MPR), support and adaptive constraints is carried out. The parameters we used for numerical simulation are as follows: the wavelength is 500 nm, the pixel pitch is 5.86 µ m, G1 creates a disk-shaped SE with a radius equal to 1 and G2 creates a 2 × 2 square SE. In-line holograms with 500 × 500 pixels are padded with constant to an image with 1000 × 1000 pixels. Given an in-line hologram with the imaging distance of 6cm, support and adaptive constraints are enforced on the object plane in the double-side constraint iteration. For the MPR method, We adopted three in-line holograms with a distance interval of 0.5 mm. These three intensity patterns are treated as amplitude constraints in the iterative phase retrieval.
In Fig. 3a1,a2, the amplitude and phase distributions of a complex-valued object with "flat" boundary serve as ground truth. The retrieved amplitude and phase by methods of support constraints, MPR and adaptive constraints are shown in Fig. 3b1-c3. Note that the reconstructions by support constraints and MPR still suffer www.nature.com/scientificreports/ from the overlapping twin-image artifact. That is because the support constraint seems as an aperture that mainly contributes to filtering artifacts beyond the object support region, but artifacts wrapped inside the pre-designed boundary can not be obstructed during iterations. MPR approach is able to achieve stable and high-fidelity reconstruction, but the reconstruction quality is restricted by the quantity of captured holograms and iterations.
In contrast with them, incorporating adaptive constrains into the iterative phase retrieval exhibits higher-quality reconstruction and better ability to tackle with the twin-image problem. The twin-image artifact is interpreted as noise terms that can be removed by morphological operations. The cross-sectional profiles match well with ground truth. In addition to reconstructing the object with stepped-phase distributions, comparison of reconstructing the object with continuous-phase distributions by means of different phase retrieval algorithms is also discussed. According to ground truth of object 2 shown in Fig. 3e1, e2, it is noticeable that the retrieved amplitude and phase using adaptive constraints still have a better performance in reconstruction quality and twin-image elimination than that using support constraints and MPR, which is depicted in Fig. 3f1-g3. Applying www.nature.com/scientificreports/ conventional support constraints generally suffers from a high reconstruction error because the loose support region is insufficient to suppress the artifact. By introducing the morphological filtering, adaptive constraints allow for gradually adjusting the support domain according to the structural characteristics of the object during iterations. The mean square errors (MSEs) E = x,y ρ x, y − ρ 0 x, y 2 / x,y ρ 0 x, y 2 of the retrieved amplitude and phase considered as a metric to reflect the reconstruction quality with different approaches are calculated in Fig. 3d1,d2,h1,h2, where ρ x, y is the estimated distribution and ρ 0 x, y is the initial distribution. It is indicated that the method of incorporating adaptive constraints is allowed to achieve a more optimized convergence performance. Another example of reconstructing a complex-valued scene with "random" boundary is depicted in Fig. 4. Figure 4a1,a2 indicate the ground-truth amplitude and phase of the original object 1. A image with sharp edges is input as the phase part. As shown in Fig. 4b1-c3, the reconstruction quality by support constraints and MPR methods is severely degraded by artifacts originating from mutual interference between the amplitude and the www.nature.com/scientificreports/ phase distributions of the object with "random" boundary. Significantly, artifacts can be completely removed by imposing adaptive constraints, which demonstrates that adaptive constraints generated by morphological operations help to accurately extract the object structure and simultaneously eliminate the unwanted information. To further visualize the reconstruction effect of applying adaptive constraints, a cell image with more complicated distribution is used as the phase part shown in Fig. 4e2. In Fig. 4f1-g3, it is observed that the use of support constraints is ineffective for retrieving the low-frequency phase information, and the reconstruction is obscured by the artifact. Enforcing adaptive constraints on the object plane appears to be a viable solution for these issues, as it enables the rendering of recognizable cellular structures. It can also be discovered in Fig. 4d1,d2,h1,h2 that adaptive constraints require significantly less time to achieve the same MSE as the other phase retrieval approaches, which illustrates that employing adaptive constraints has higher convergence speed and lower MSE. Therefore, adaptive constraints incorporated into the phase iteration method have improvement in fidelity of reconstruction and convergence behavior.
Reconstruction under noisy conditions. Furthermore, single-shot phase retrieval inherits ill-posedness, which makes it susceptible to measurement noise. In order to evaluate the noise immunity of adaptive constraints implemented in the iterative phase retrieval, the simulated hologram is added white Gaussian noise with different signal to noise ratio (SNR). Evolutions of adaptive constraints when white Gaussian noise with the SNR of 15 dB is added to the simulated hologram are described in Fig. 5a. It is noteworthy that adaptive constraints updated automatically at each iteration contribute to sketching the contours of the object and filtering the reconstruction immersed in noise. Support constraints designed according to the object shape help to reduce noise outside the known boundary, but noise superimposed on the object within the support region still degrades the reconstruction quality, as seen in Fig. 5c1,c2. As depicted in Fig. 5b1,b2, reconstructions by adaptive constraints www.nature.com/scientificreports/ achieve a better recovery performance than using support constraints. To quantify the convergence behavior under noisy conditions, plot of logarithm MSE against iterations is performed in Fig. 5d. The reconstruction has a lower MSE with the increase of SNR. It can be observed that the MSE enlarges with the increasing of iterations by means of support constraints. That is because the noise within the support domain cannot be filtered out and accumulates during the iteration process, resulting in an increased reconstruction error. In contrast, adaptive constraints ensure convergence under moderate-noise conditions, having a better noise tolerance. Thus, using adaptive constraints is profitable to enhance the reconstruction quality and improve convergence behavior under noisy conditions.

Experimental reconstruction
Next, we go further to experimentally confirm the improved reconstruction performance of our proposed method by imaging an amplitude resolution test target. By methods of nano processing, the positive 1951 USAF target (Thorlabs, Inc.) is fabricated by etching lines onto a glass plate. The raw in-line hologram in Fig. 6a is measured by the CMOS sensor at the distance of 4.14 cm from the object. For MPR method, the distance interval between three hologram planes is 0.5 mm. With the increasing of iterations, evolutions of adaptive constraints in Fig. 6b illustrate that introducing morphological operations helps to probe the structural features of the object to adaptively form a tight and sharp object support. Comparisons of the amplitude reconstruction by applying support constraints, MPR and adaptive constraints are performed in Fig. 6c1-d3. The retrieved amplitude by support constraints and MPR is severely obscured due to the existence of high reconstruction noise. By contrast, adaptive constraints can accomplish higher-quality reconstruction without artifacts overlapped, exhibiting better  www.nature.com/scientificreports/ resolution with line width ranging from 8.77 to 15.63 µ m (Group 5, Elements 1-6). Also, Group 6, Element 3 of USAF 1951 with linewidth 6.2 µ m is the finest resolvable feature, which can be found in Fig. 6d3. It can be elucidated that measurement noise and the twin-image artifact which is interpreted as noise can both be removed by morphological operations. In addition, we imaged a phase plate to demonstrate the quantitative phase imaging ability of incorporating adaptive constraints. The phase plate is fabricated by etching multiple binary patterns onto a quartz glass plate. The raw in-line hologram in Fig. 7a is captured at the distance of 4.31 cm from the object. As observed in Fig.7b-d which present the surface profiles and the cross-sectional profiles of the retrieved phase by different approaches, the reconstruction quality by support constraints and MPR method is degraded attributing to the wrap-around artifact. Whereas enforcing adaptive constraints on the object domain is profit for eliminating twin-image artifacts from the retrieved phase distribution effectively. Besides, the low-frequency phase can hardly transfer into the intensity at the sensor plane because the frequency response of the weak phase transfer function declines to zero 60 , which poses challenges for recovering the low-frequency phase based on the in-line holographic system. It is noticed that iteratively updating adaptive constraints at the object plane seems to address this problem, which has a efficient reconstruction of the phase object. The experimental reconstruction confirms the improvement of reconstruction performance and the effectiveness in removing artifacts by exploiting adaptive constraints.
Furthermore, the skeletal muscle cells sandwiched between glass plates are selected to demonstrate the capacity of our proposed method to reconstruct complex-valued objects. Figure 8a shows the measured in-line holograms. Notice that the retrieved amplitude in Fig. 8b1-b3 and the retrieved phase in Fig. 8c1-c3 by methods of adaptive constraints present a better reconstruction than using support constraints and MPR. Using adaptive constraints can remove twin-image artifacts effectively and reveal the cell morphology clearly. For cell samples, too little light scattering from the specimen commonly results in a poor image contrast, which is difficult to bring out a distinguishable structure from an overwhelming incident light background 8 . The reconstructions performed in Fig. 8b3,c3 suggest that imposing adaptive constraints to confine the object support can significantly improve the image contrast and resolution.

Conclusion
In summary, we propose an advanced iterative phase retrieval framework for single-shot in-line holographic imaging that incorporates adaptive constraints, which achieves high-fidelity reconstruction and optimizes convergence performance. Different from previous strategies for generating the object support, adaptive constraints by introducing morphological operations enable the object support to be automatically and accurately updated at each iteration. The capability of morphological filtering technique to extract the object structure while eliminating the irrelevant part provides a more appropriate and efficient estimation of support. Reconstruction of complex-valued objects by employing adaptive constraints is investigated and the immunity to noise is demonstrated. Compared with applying support constraints and MPR method, adaptive constraints  The retrieved amplitude by using support constraints, MPR and adaptive constraints respectively after 45 iterations. (c1-c3) The retrieved phase by using support constraints, MPR and adaptive constraints respectively after 45 iterations.